Read the original article here.

---

Virtual Keyboards: An Innovation Ahead of Its Time

Introduction

The virtual keyboard, a software component that allows users to input characters without physical keys, stands as a testament to early innovation in computer interface design. Conceived long before the ubiquitous touchscreens of today, the virtual keyboard was a visionary solution to input challenges in a world where physical keyboards were not always practical or accessible. Initially developed for users with disabilities and multilingual typists, its true potential was realized with the advent of mobile computing and touch-sensitive displays. This exploration delves into the history, types, and evolution of the virtual keyboard, highlighting its status as a computer innovation that was truly ahead of its time.

Types of Virtual Keyboards

Virtual keyboards are not a monolithic entity; they encompass a variety of designs and technologies tailored to different needs and contexts. Here are some key types:

Touchscreen Virtual Keyboards

The most prevalent type today, touchscreen virtual keyboards are displayed directly on a device's screen. Users interact with them by tapping or tracing their fingers across the virtual keys.

Touchscreen: A display technology that allows a user to interact with a computer by touching areas on the screen. Touchscreens can detect the presence and location of a touch within the display area.

These keyboards are software-based and highly adaptable. They can be customized with different layouts, languages, and features.

Examples and Use Cases:

• Smartphones and Tablets: The primary input method for most modern mobile devices.
• Point-of-Sale Systems: Used in retail and hospitality for efficient and space-saving data entry.
• Interactive Kiosks: Public terminals for information access or transactions, often utilizing virtual keyboards for user input in a space-constrained environment.

Optically Projected Virtual Keyboards

This innovative approach projects the image of a keyboard onto a flat surface. Sensors detect finger movements as users "type" on the projected keys.

Optical Virtual Keyboard: A keyboard system that projects a keyboard layout onto a surface and uses optical sensors to detect finger movements and interpret them as keystrokes.

This technology eliminates the need for any physical keyboard, creating an input device out of thin air.

Examples and Use Cases:

• Early Mobile Devices and PDAs: Offered as a compact solution for text input on devices with limited space.
• Presentations and Public Speaking: Allows presenters to control computers or devices from a distance without a physical keyboard.
• Clean Room Environments: Suitable for sterile environments like hospitals or laboratories where physical keyboards can be difficult to sanitize or introduce contaminants.

Optically Detected Hand and Finger Motion Keyboards

This type of virtual keyboard takes a different approach, directly tracking hand and finger movements without projecting a keyboard layout. Cameras or sensors analyze hand gestures and translate them into text input.

Examples and Use Cases:

• Gesture-Based Computing: An early exploration into gesture recognition as an input method, foreshadowing modern gesture controls in gaming and virtual reality.
• Assistive Technology: Potentially useful for users with motor impairments who can use hand gestures for communication.
• Research and Development: Primarily a research concept, this type showcases early attempts to move beyond traditional keyboard layouts and explore more natural input methods.

Virtual Chorded Keyboards

Chorded keyboards are designed for one-handed operation, where users press combinations of keys (chords) to input different characters or commands. Virtual chorded keyboards bring this concept to touchscreens.

Chorded Keyboard: A computer input device that allows the user to enter characters or commands by pressing combinations of two or more keys together.

These keyboards can be more efficient for experienced users once they learn the chord combinations, especially in mobile or wearable contexts.

Examples and Use Cases:

• Wearable Computing: Small and efficient input for devices like smartwatches or early wearable computers.
• Accessibility: Can be adapted for users with single-handed dexterity.
• Specialized Applications: Potentially useful in niche applications where rapid, one-handed input is required.

Virtual Keyboards in Mobile Devices

The rise of mobile devices, particularly PDAs and smartphones, was a crucial catalyst for the development and widespread adoption of virtual keyboards. These devices, by their very nature, lacked the physical space for traditional keyboards, making virtual solutions essential.

Historical Development: From PDA Limitations to iPhone Breakthrough

PDA Era: Early Challenges

Personal Digital Assistants (PDAs) in the late 1990s and early 2000s were early forerunners of smartphones, requiring text input for various functions. The primary input methods explored were:

• Stylus-Operated Virtual Keyboards: Users would tap on-screen keys with a stylus.
• External USB Keyboards: Offered a familiar typing experience but negated the portability of PDAs.
• Handwritten Character Recognition: Translating handwritten input into text.
• Stroke Recognition (e.g., Graffiti): Simplified handwriting systems that used stylized strokes for characters.

Microsoft's mobile operating systems of the time attempted to replicate full-sized keyboards virtually, leading to cramped and difficult-to-use layouts. A major limitation was the lack of multi-touch technology.

Multi-touch Technology: A touchscreen capability that allows a device to recognize and respond to multiple points of contact simultaneously.

Without multi-touch, PDAs struggled with virtual keyboards because:

• Accuracy Issues: It was difficult to accurately tap small keys with a stylus, leading to frequent errors.
• Slow Typing Speed: The lack of tactile feedback and precision made typing slow and frustrating.
• Usability Constraints: The overall experience was far from user-friendly compared to physical keyboards.

The First iPhone: A Multi-Touch Revolution

When Apple introduced the iPhone in 2007, the absence of a physical keyboard was initially met with skepticism. However, Apple’s integration of multi-touch technology was a game-changer. Multi-touch allowed for:

• More Accurate Typing: The system could differentiate between intentional taps and accidental touches, significantly improving accuracy.
• Gestures and Enhanced Input: Multi-touch paved the way for gestures like pinch-to-zoom and also allowed for more sophisticated virtual keyboard interactions in the future.
• Improved User Experience: The iPhone's virtual keyboard was surprisingly usable, overcoming the limitations of previous PDA keyboards and setting a new standard for mobile input.

The iPhone demonstrated that with the right technology, virtual keyboards could be a viable and even superior input method for mobile devices, marking a turning point in their development.

Current Implementation and Use

Today, virtual keyboards are ubiquitous across mobile operating systems and continue to evolve with advanced features.

Android

Android's architecture provides a flexible framework for virtual keyboards through the "InputMethodService" within the Android SDK (Software Development Kit).

Software Development Kit (SDK): A set of software development tools that allows the creation of applications for a certain software package, software framework, hardware platform, computer system, video game console, operating system, or similar development platform.

This service allows developers to create custom keyboard layouts and input methods. The "KeyboardView" component and "Keyboard Class" aid in defining the visual layout and key attributes of a virtual keyboard using XML descriptions.

This openness has led to a vibrant ecosystem of third-party keyboard apps for Android, such as Gboard and SwiftKey, offering users a wide range of features and customizations. Users can easily switch between different keyboards through the Android settings menu.

iOS

Apple's iOS also supports custom virtual keyboards, though with some restrictions compared to Android. Developers can utilize the "UIInputViewController" to build custom keyboard extensions. However, iOS provides limited access to system-level features like the device dictionary or general keyboard settings. iOS manages the switching between system and custom keyboards automatically when a user interacts with a text input field.

Windows

Microsoft Windows provides a built-in virtual keyboard through the "Common Text Framework" service, primarily for accessibility and touchscreen devices. This virtual keyboard offers basic functionality and integration within the Windows environment.

Word Suggestions and Predictive Text

Long before the smartphone era, research in the early 2000s highlighted the potential of word prediction to enhance typing speed.

Word Prediction (Predictive Text): A feature that suggests words or phrases as a user types, aiming to speed up input and reduce typing effort.

Early implementations relied on static dictionaries. However, advancements in algorithms and data analysis led to more sophisticated predictive text features.

• 2013 (Android 4.4): Google integrated predictive text into Android, driven by the innovations of third-party keyboard providers like SwiftKey and Swype.
• 2014 (iOS 8): Apple introduced "QuickType," a predictive typing feature displaying word suggestions above the keyboard.

Predictive text has become a standard feature in modern virtual keyboards, significantly improving typing efficiency, especially on mobile devices.

Haptic Feedback

Haptic feedback adds a tactile dimension to virtual keyboards, providing users with a physical sensation when they press keys.

Haptic Feedback: Technology that uses tactile sensations to communicate information to the user. In virtual keyboards, this often involves vibrations or simulated clicks when keys are pressed.

This feedback is crucial for improving the typing experience on touchscreens, making it feel more like typing on a physical keyboard. Early research in 2004, as documented in "A Comparison of Two Input Methods for Keypads on Mobile Devices," explored the use of hysteresis in haptic feedback.

Hysteresis (in Haptic Feedback): In the context of virtual keyboards, hysteresis refers to creating a more realistic "key press" feel by providing an initial "click" sensation when a key is pressed down, followed by a separate "unclick" sensation when the finger pressure is reduced, mimicking the behavior of a physical key returning to its resting state.

Haptic feedback, especially with hysteresis, greatly enhances the usability and satisfaction of virtual keyboards by providing crucial tactile confirmation of key presses.

Special Keyboard Types: Expanding the Virtual Realm

Virtual keyboards are not limited to touchscreens. They have been adapted for emerging technologies like augmented reality (AR) and virtual reality (VR), and explored in innovative forms like optical virtual keyboards.

Optical Virtual Keyboard: The Invisible Interface

Invented and patented by IBM engineers in 1992, the optical virtual keyboard was a remarkable innovation. It utilized optical sensors to detect hand and finger movements and interpret them as input on a non-existent keyboard.

This technology could emulate various manual input devices, including keyboards and mice, offering flexibility and adaptability for different applications. Samsung's "Selfie Type," showcased in 2020, was an attempt to bring this concept to smartphones, using the front-facing camera to track fingers on an "invisible keyboard." While "Selfie Type" didn't launch commercially, it demonstrates the enduring appeal of optical virtual keyboard concepts.

Augmented Reality (AR) Keyboards: Blending Virtual and Real

In augmented reality, virtual keyboards provide a means of text input within the mixed reality environment. Common approaches include:

• Rendering a Flat Virtual Keyboard: Projecting a 2D keyboard into the user's AR view, often using libraries like Unity's TouchScreenKeyboard.
• Head-Gaze Input: Using head movements to point and select keys, as implemented in Microsoft HoloLens.
• Wearable Finger Tracking: Early research, like that by the Korean KJIST U-VR Lab in 2003, explored using wearables to track finger movements and provide audiovisual feedback for a more natural typing experience in AR.

The Magic Leap 1 AR headset incorporated a virtual keyboard, demonstrating the practical application of AR keyboards in commercial products.

Virtual Reality (VR) Keyboards: Text Input in Immersive Worlds

Virtual reality environments present unique challenges for text input, as users are fully immersed and often their hands are not tracked by default. Solutions include:

• Controller-Based Pointing: VR systems like Google Daydream (2016) used controllers to point at virtual keys on a rendered keyboard.
• Hand Tracking Systems: More advanced VR systems and experimental setups, like Logitech's BRIDGE developers kit (2017) with Vive Trackers and Logitech G gaming keyboards, aimed to track finger movements directly, allowing users to type on physical keyboards that are represented in VR.

These VR keyboard solutions strive to provide intuitive text input methods within immersive virtual experiences.

Security Considerations

While virtual keyboards can offer some security advantages, they also introduce their own set of security concerns.

Mitigating Keystroke Logging

Virtual keyboards were initially adopted in security-sensitive contexts, like online banking (e.g., Westpac) and government financial services (e.g., TreasuryDirect), to counter keystroke logging.

Keystroke Logging: The act of recording the keys pressed on a keyboard, typically without the user's knowledge. Malware can be used to log keystrokes and steal sensitive information like passwords.

By using a virtual keyboard and clicking keys with a mouse, it becomes more difficult for traditional keystroke logging malware to capture the input, as no physical keys are pressed. However, this is not a foolproof solution.

Vulnerabilities of Virtual Keyboards

Virtual keyboards are still susceptible to various attacks:

• Screen Capture: Malware can take screenshots at regular intervals or upon mouse clicks, capturing the virtual keyboard and the selected keys.
• Shoulder Surfing: Observing the screen directly to see which virtual keys are being clicked can be easier than observing a physical keyboard.
• Visual Feedback Exploitation: If the virtual keyboard provides strong visual feedback (e.g., color changes, focus highlighting) when keys are clicked, it becomes easier for an observer to decipher the input.
• Telemetry and Data Leakage: Some virtual keyboards may collect usage data (telemetry) or leak sensitive information through text suggestions, potentially compromising privacy.
• Keystroke Inference Attacks: While designed to prevent direct keystroke logging, sophisticated attacks might still infer input patterns based on mouse movements or other data.

Therefore, while virtual keyboards can add a layer of security against certain threats, they are not a complete security solution and should be used in conjunction with other security measures.

Conclusion

The virtual keyboard, initially conceived as an alternative input method for specific user groups and situations, has evolved into a cornerstone of modern computing, particularly in the mobile era. From early stylus-based implementations on PDAs to the sophisticated multi-touch keyboards of smartphones and tablets, and even extending into the realms of AR and VR, the virtual keyboard's journey is a compelling example of an innovation that was indeed ahead of its time. Its development was driven by the need to overcome the limitations of physical keyboards in new computing contexts, and its continued evolution showcases the ongoing quest for more intuitive, versatile, and secure input methods in an increasingly digital world. The virtual keyboard's story is a reminder that sometimes the most impactful innovations are those that anticipate future needs and lay the groundwork for technologies that we now consider indispensable.